Activation of remote devices in a networked system

ABSTRACT

The present disclosure is generally directed to the generation of voice-activated data flows in interconnected network. The voice-activated data flows can include input audio signals that include a request and are detected at a client device. The client device can transmit the input audio signal to a data processing system, where the input audio signal can be parsed and passed to the data processing system of a service provider to fulfill the request in the input audio signal. The present solution is configured to conserve network resources by reducing the number of network transmissions needed to fulfill a request.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 120 asa continuation of U.S. patent application Ser. No. 16/107,617, filedAug. 21, 2018, which claims the benefit of priority under 35 U.S.C. §120 as a continuation of U.S. patent application Ser. No. 16/064,961filed Jun. 21, 2018, which is a U.S. National Stage under 35 U.C.S. §371 of International Patent Application No. PCT/US2018/031451, filed onMay 7, 2018 and designating the United States. The foregoingapplications are hereby incorporated by reference in their entirety.

BACKGROUND

Excessive network transmissions, packet-based or otherwise, of networktraffic data between computing devices can prevent a computing devicefrom properly processing the network traffic data, completing anoperation related to the network traffic data, or timely responding tothe network traffic data. The excessive network transmissions of networktraffic data can also complicate data routing or degrade the quality ofthe response if the responding computing device is at or above itsprocessing capacity, which may result in inefficient bandwidthutilization. The excessive network transmissions can occur when multiplenetwork transmissions are used to clarify or request additionalinformation in response to a first network transmission.

SUMMARY

According to at least one aspect of the disclosure, a system to generatevoice-activated threads in a networked computer environment can includea data processing system. The data processing system can include one ormore processors and memory. The data processing system can execute anatural language processor (“NLP”), a remote application launcher, andan action handler component. The data processing system can receive, bythe NLP component, a first input audio signal detected by a sensor of afirst client computing device. The data processing system can parse, bythe NLP component, the first input audio signal to identify a firstrequest and a first entity. The data processing system can determine, bythe action handler component, a plurality of candidate service providerdevices. Each of the plurality of candidate service provider devices canbe configured to fulfill the first request. Each of the plurality ofcandidate service provider device can be associated with a respectiveapplication installed on the first client computing device. The dataprocessing system can select, by the action handler component, a serviceprovider device from the plurality of candidate service providerdevices. The data processing system can generate, by the remoteapplication launcher, a digital component. The digital component caninclude an indication of the application associated with the serviceprovider device and the first entity. The digital component can beconfigured to launch the application associated with the serviceprovider device when executed by the first client computing device tofulfill the first request. The data processing system can transmit, bythe action handler component, the digital component to the first clientcomputing device in response to the first input audio signal.

According to at least one aspect of the disclosure, a method to generatevoice-activated threads in a networked computer environment can includereceiving, by an NLP component executed by a data processing system, afirst input audio signal detected by a sensor of a first clientcomputing device. The method can include parsing, by the NLP component,the first input audio signal to identify a first request and a firstentity. The method can include determining, by an action handlercomponent executed by the data processing system, a plurality ofcandidate service provider devices. Each of the plurality of candidateservice provider devices can be configured to fulfill the first request.Each of the plurality of candidate service provider device can beassociated with a respective application installed on the first clientcomputing device. The method can include selecting, by the actionhandler component, a service provider device from the plurality ofcandidate service provider devices. The method can include generating,by a remote application launcher executed by the data processing system,a digital component. The digital component can include an indication ofthe application associated with the service provider device and thefirst entity. The digital component can be configured to launch theapplication associated with the service provider device when executed bythe first client computing device to fulfill the first request. Themethod can include transmitting, by the action handler component, thedigital component to the first client computing device in response tothe first input audio signal.

According to at least one aspect of the disclosure, a system to generatevoice-activated threads in a networked computer environment can includea data processing system. The data processing system can execute an NLPcomponent and an action handler component. The data processing systemcan receive a first input audio signal that is detected by a sensor of afirst client computing device. The data processing system can parse thefirst input audio signal to identify a first request. The dataprocessing system can select a first action data structure based on thefirst request. The first action data structure can be associated with afirst service provider device. The data processing system can transmit afirst audio-based input request to the first client based at least on afield in the first action data structure. The data processing system canreceive a second input audio signal detected by the sensor of the firstclient computing device that is generated in response to the firstaudio-based input request. The data processing system can parse thesecond input audio signal to identify a response entity in the secondinput audio signal. The data processing system can expand the responseentity based on an expansion policy that is associated with the firstclient computing device. Expanding the response entity can includeconverting the response entity into format associated with the field inthe first action data structure. The data processing system can populatethe expanded response entity into the field of the first action datastructure. The data processing system can transmit the first action datastructure to the first service provide to fulfill the first request.

According to at least one aspect of the disclosure, a method to generatevoice-activated threads in a networked computer environment can includereceiving a first input audio signal that is detected by a sensor of afirst client computing device. The method can include parsing the firstinput audio signal to identify a first request. The method can includeselecting a first action data structure based on the first request. Thefirst action data structure can be associated with a first serviceprovider device. The method can include transmitting a first audio-basedinput request to the first client based at least on a field in the firstaction data structure. The method can include receiving a second inputaudio signal detected by the sensor of the first client computing devicethat is generated in response to the first audio-based input request.The method can include parsing the second input audio signal to identifya response entity in the second input audio signal. The method caninclude expanding the response entity based on an expansion policy thatis associated with the first client computing device. Expanding theresponse entity can include converting the response entity into formatassociated with the field in the first action data structure. The methodcan include populating the expanded response entity into the field ofthe first action data structure. The method can include transmitting thefirst action data structure to the first service provide to fulfill thefirst request.

Each aspect can optionally include one or more of the followingfeatures. Parsing, by the natural language processor component, thefirst input audio signal to identify a second entity; expanding, by theaction handler component executed by the data processing system, thesecond entity based on an expansion policy associated with the firstclient computing device and into a format associated with a field of thedigital component; generating, by the action handler component, thedigital component to include the expanded second entity in the field ofthe digital component. Parsing, by the natural language processorcomponent, the first input audio signal to identify a second entity; anddetermining, by the action handler component, the second entity cannotbe expanded based on an expansion policy associated with the firstclient computing device. Transmitting, by the action handler component,an audio-based input request to the first client computing device torequest an updated entity. The digital component may be configured topopulate a field of the application associated with the service providerdevice with the first entity when executed by the first client computingdevice. Measuring, by the action handler component, a network latencybetween the data processing system and each of the plurality ofcandidate service provider devices; and selecting, by the action handlercomponent, the service provider device from the plurality of candidateservice provider devices based at least on the network latency betweenthe data processing system and each of the plurality of candidateservice provider devices. Selecting, by the action handler component,the service provider device from the plurality of candidate serviceprovider devices based at least a performance of each of the pluralityof candidate service provider devices. Selecting, by the action handlercomponent, the service provider device from the plurality of candidateservice provider devices based at least a preference associated with thefirst client computing device. Receiving, by the natural languageprocessor component, a second input audio signal detected by the sensorof the first client computing device, the second input audio signalcomprising an indication of the service provider device; and selecting,by the action handler component, the service provider device from theplurality of candidate service provider devices based at least on theindication of the service provider device. Receiving, by the naturallanguage processor component, a second input audio signal detected bythe sensor of the first client computing device; parsing, by the naturallanguage processor component, the second input audio signal to identifya second request and a second entity; determining, by the action handlercomponent, the service provider device is configured to fulfill thesecond request; generating, by the action handler component, a firstaction data structure based on the second request and comprising thesecond entity; and transmitting, by the action handler component, thefirst action data structure to the service provider device.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations andprovide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component may be labeled inevery drawing. In the drawings:

FIG. 1 illustrates block diagram of an example system to authenticatecomputing devices, in accordance with an example of the presentdisclosure.

FIG. 2 illustrates a block diagram of an example method to generatevoice-activated threads in the system illustrated in FIG. 1, inaccordance with an example of the present disclosure.

FIG. 3 illustrates a block diagram of example data flows betweencomponents of the system illustrated in FIG. 1, in accordance with anexample of the present disclosure.

FIG. 4 illustrates a block diagram of an example method to generatevoice-activated threads in the system illustrated in FIG. 1, inaccordance with an example of the present disclosure.

FIG. 5 illustrates a block diagram of example data flows betweencomponents of the system illustrated in FIG. 1, in accordance with anexample of the present disclosure.

FIG. 6 is a block diagram of an example computer system that can be usedin the system illustrated in FIG. 1, in accordance with an example ofthe present disclosure.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various conceptsrelated to, and implementations of, methods, apparatuses, and systems togenerate voice-activated data flows in interconnected network. Thevarious concepts introduced above and discussed in greater detail belowmay be implemented in any of numerous ways.

The present disclosure is generally directed to the generation ofvoice-activated data flows in interconnected networks. Thevoice-activated data flows can include input audio signals that includea request and are detected at a client device. The client device cantransmit the input audio signal to a data processing system, where theinput audio signal can be parsed and passed to the data processingsystem of a service provider to fulfill the request in the input audiosignal. Voice-activated systems can result in excessive datatransmissions because one or more of the data processing systems may notbe able to parse or understand the entities or terms in the input audiosignals. For example, because of the conversational nature of thevoice-activated system, users can often provide conversational-styleanswers that can be vague or unclear when removed from context. Forexample, when placing an order with a voice-driven (and hands free)digital assistant device, the data processing system of the serviceprovider may receive an input audio signal from the digital assistantdevice that includes the statement “order a coffee from the coffee shopnear my house.” The service provider's data processing system mayunderstand that the phrase “near my house” is a location entity;however, the service provider's data processing system may not be ableto convert the phrase into a specific location or the address of aspecific coffee shop. This can result in wasted computational resourcesas the service provider's data processing system transmits follow uprequests to the digital assistant device to request clarifying oradditional input audio signals.

Systems and methods of the present technical solution enable a reductionin reducing computing resource utilization, and network bandwidth byreducing the number of network transmissions required to completevoice-based requests. The present solution can enable the digitalassistant to automatically convert unclear terms into terms that can beprocessed by the service provider's data processing system, which canreduce the number of follow up input audio signals required to completea request. For example, the present solution can enable the digitalassistant to convert the term “home” into a specific address prior totransmitting the term to the service provider's data processing system,such that “123 Main St.” can be transmitted to the service provider'sdata processing system rather than the term “home.”

The present solution can also increase the privacy of the end user. Forexample, the end user can approve of transmitting an address to theservice provider, a third-party. However, the end user may not wish toallow the third-party to know that the provided location is the enduser's home address. The present solution can enable the digitalassistant to convert the term “home” (or other protected term) intoformatted data (e.g., a specific address), which the third-party doesnot know is the user's home. The present solution can address problemsassociated with third-party service providers interacting with a userusing a voice-based interface and can overcome technical challengesassociated with allowing third-party service providers to operatetogether with a voice-based system.

FIG. 1 illustrates an example system 100 to generate voice-activatedthreads in a networked computer environment. The system 100 can includea data processing system 102. The data processing system 102 cancommunicate with one or more of a digital component provider device 106(e.g., content provider device), client computing devices 104, orservice provider devices 160 via a network 105. The network 105 caninclude computer networks such as the Internet, local, wide, metro, orother area networks, intranets, satellite networks, and othercommunication networks such as voice or data mobile telephone networks.The network 105 can be used to access information resources such as webpages, web sites, domain names, or uniform resource locators that can bepresented, output, rendered, or displayed on at least one computingdevice 104, such as a laptop, desktop, tablet, digital assistant,personal digital assistant, smartwatch, wearable device, smart phone,portable computer, or speaker. For example, via the network 105 a userof the client computing device 104 can access information, data, orservices provided by digital component provider devices 106 or theservice provider devices 160. The client computing device 104 may or maynot include a display. For example, the client computing device 104 mayinclude limited types of user interfaces, such as a microphone andspeaker (e.g., the client computing device 104 can include a voice-driveor audio-based interface). The primary user interface of the computingdevice 104 can include a microphone and speaker.

The network 105 can include or constitute a display network, e.g., asubset of information resources available on the internet that areassociated with a content placement or search engine results system, orthat are eligible to include third party digital components. The network105 can be used by the data processing system 102 to access informationresources such as web pages, web sites, domain names, or uniformresource locators that can be presented, output, rendered, or displayedby the client computing device 104. For example, via the network 105 auser of the client computing device 104 can access information or dataprovided by the digital component provider device 106.

The network 105 may be any type or form of network and may include anyof the following: a point-to-point network, a broadcast network, a widearea network, a local area network, a telecommunications network, a datacommunication network, a computer network, an ATM (Asynchronous TransferMode) network, a SONET (Synchronous Optical Network) network, a SDH(Synchronous Digital Hierarchy) network, a wireless network, and awireline network. The network 105 may include a wireless link, such asan infrared channel or satellite band. The topology of the network 105may include a bus, star, or ring network topology. The network mayinclude mobile telephone networks using any protocol or protocols usedto communicate among mobile devices, including advanced mobile phoneprotocol (“AMPS”), time division multiple access (“TDMA”), code-divisionmultiple access (“CDMA”), global system for mobile communication(“GSM”), general packet radio services (“GPRS”), or universal mobiletelecommunications system (“UMTS”). Different types of data may betransmitted via different protocols, or the same types of data may betransmitted via different protocols.

The system 100 can include at least one data processing system 102. Thedata processing system 102 can include at least one logic device such asa computing device having a processor to communicate via the network105, for example, with the computing device 104, service provider device160, or the digital component provider device 106. The data processingsystem 102 can include at least one computation resource, server,processor, or memory. For example, the data processing system 102 caninclude a plurality of computation resources or servers located in atleast one data center. The data processing system 102 can includemultiple, logically-grouped servers and facilitate distributed computingtechniques. The logical group of servers may be referred to as a datacenter, server farm, or a machine farm. The servers can also begeographically dispersed. A data center or machine farm may beadministered as a single entity, or the machine farm can include aplurality of machine farms. The servers within each machine farm can beheterogeneous: one or more of the servers or machines can operateaccording to one or more type of operating system platform.

Servers in the machine farm can be stored in high-density rack systems,along with associated storage systems, and located in an enterprise datacenter. For example, consolidating the servers in this way may improvesystem manageability, data security, the physical security of thesystem, and system performance by locating servers and high-performancestorage systems on localized high-performance networks. Centralizationof all or some of the data processing system 102 components, includingservers and storage systems, and coupling them with advanced systemmanagement tools, allows more efficient use of server resources, whichsaves power and processing requirements and reduces bandwidth usage.

The client computing device 104 can include, execute, interface, orotherwise communicate with one or more of at least one local digitalassistant 134, at least one sensor 138, at least one transducer 140, atleast one audio driver 142, or at least one display 144. The sensor 138can include, for example, a camera, an ambient light sensor, proximitysensor, temperature sensor, accelerometer, gyroscope, motion detector,GPS sensor, location sensor, microphone, video, image detection, ortouch sensor. The transducer 140 can include or be part of a speaker ora microphone. The audio driver 142 can provide a software interface tothe hardware transducer 140. The audio driver 142 can execute the audiofile or other instructions provided by the data processing system 102 tocontrol the transducer 140 to generate a corresponding acoustic wave orsound wave. The display 144 can include one or more hardware or softwarecomponent configured to provide a visual indication or optical output,such as a light emitting diode, organic light emitting diode, liquidcrystal display, laser, or display.

The local digital assistant 134 can include or be executed by one ormore processors, logic array, or memory. The local digital assistant 134can detect a keyword and perform an action based on the keyword. Thelocal digital assistance 134 can be an instance of the remote digitalassistance component 112 executed at the data processing system 102 orcan perform any of the functions of the remote digital assistancecomponent 112. The local digital assistant 134 can filter out one ormore terms or modify the terms prior to transmitting the terms as datato the data processing system 102 (e.g., remote digital assistantcomponent 112) for further processing. The local digital assistant 134can convert the analog audio signals detected by the transducer 140 intoa digital audio signal and transmit one or more data packets carryingthe digital audio signal to the data processing system 102 via thenetwork 105. The local digital assistant 134 can transmit data packetscarrying some or all of the input audio signal responsive to detectingan instruction to perform such transmission. The instruction caninclude, for example, a trigger keyword or other keyword or approval totransmit data packets comprising the input audio signal to the dataprocessing system 102.

The local digital assistant 134 can perform pre-filtering orpre-processing on the input audio signal to remove certain frequenciesof audio. The pre-filtering can include filters such as a low-passfilter, high-pass filter, or a bandpass filter. The filters can beapplied in the frequency domain. The filters can be applied usingdigital signal processing techniques. The filter can be configured tokeep frequencies that correspond to a human voice or human speech, whileeliminating frequencies that fall outside the typical frequencies ofhuman speech. For example, a bandpass filter can be configured to removefrequencies below a first threshold (e.g., 70 Hz, 75 Hz, 80 Hz, 85 Hz,90 Hz, 95 Hz, 100 Hz, or 105 Hz) and above a second threshold (e.g., 200Hz, 205 Hz, 210 Hz, 225 Hz, 235 Hz, 245 Hz, or 255 Hz). Applying abandpass filter can reduce computing resource utilization in downstreamprocessing. The local digital assistant 134 on the computing device 104can apply the bandpass filter prior to transmitting the input audiosignal to the data processing system 102, thereby reducing networkbandwidth utilization. However, based on the computing resourcesavailable to the computing device 104 and the available networkbandwidth, it may be more efficient to provide the input audio signal tothe data processing system 102 to allow the data processing system 102to perform the filtering.

The local digital assistant 134 can apply additional pre-processing orpre-filtering techniques such as noise reduction techniques to reduceambient noise levels that can interfere with the natural languageprocessor. Noise reduction techniques can improve accuracy and speed ofthe natural language processor, thereby improving the performance of thedata processing system 102 and manage rendering of a graphical userinterface provided via the display 144.

The client computing device 104 can be associated with an end user thatenters voice queries as audio input into the client computing device 104(via the sensor 138 or transducer 140) and receives audio (or other)output from the data processing system 102 or digital component providerdevice 106 to present, display, or render to the end user of the clientcomputing device 104. The digital component can include acomputer-generated voice that can be provided from the data processingsystem 102 or digital component provider device 106 to the clientcomputing device 104. The client computing device 104 can render thecomputer-generated voice to the end user via the transducer 140 (e.g., aspeaker). The computer-generated voice can include recordings from areal person or computer-generated language. The client computing device104 can provide visual output via a display device 144 communicativelycoupled to the computing device 104.

The end user that enters the voice queries to the client computingdevice 104 can be associated with multiple client computing devices 104.For example, the end user can be associated with a first clientcomputing device 104 that can be a speaker-based digital assistantdevice, a second client computing device 104 that can be a mobile device(e.g., a smartphone), and a third client computing device 104 that canbe a desktop computer. The data processing system 102 can associate eachof the client computing devices 104 through a common login, location,network, or other linking data. For example, the end user may log intoeach of the client computing devices 104 with the same account user nameand password.

The client computing device 104 can receive an input audio signaldetected by a sensor 138 (e.g., microphone) of the computing device 104.The input audio signal can include, for example, a query, question,command, instructions, or other statement provided in a spoken language.The input audio signal can include an identifier or name of athird-party (e.g., a digital component provider device 106) to which thequestion or request is directed. For example, the query can include thename of the subscription-based music service (an example digitalcomponent provider device 106) in the input audio signal in order toinstruct the data processing system 102 to provide the request to thespecified subscription-based music service. For example, the input audiosignal can include “Play my music playlist on XYZ Music Service.” Themusic service can provide the songs associated with the playlist to theclient computing device 104 through the network 105 or to the dataprocessing system 102, which can provide the songs associated with theplaylist to the client computing device 104 through the network 105.

The client computing device 104 can include, execute, or be referred toas a digital assistant device. The digital assistant device can includeone or more components of the computing device 104. The digitalassistant device can include a graphics driver that can receive displayoutput from the data processing system 102 and render the display outputon display 132. The graphics driver can include hardware or softwarecomponents that control or enhance how graphics or visual output isdisplayed on the display 144. The graphics driver can include, forexample, a program that controls how the graphic components work withthe rest of the computing device 104 (or digital assistant). The localdigital assistant 134 can filter the input audio signal to create afiltered input audio signal, convert the filtered input audio signal todata packets, and transmit the data packets to a data processing systemcomprising one or more processors and memory.

The digital assistant device can include an audio driver 142 and aspeaker component (e.g., transducer 140). The pre-processor component140 can receive an indication of the display output and instruct theaudio driver 142 to generate an output audio signal to cause the speakercomponent (e.g., transducer 140) to transmit an audio outputcorresponding to the indication of the display output.

The system 100 can include, access, or otherwise interact with at leastdigital component provider device 106. The digital component providerdevice 106 can include one or more servers that can provide digitalcomponents to the client computing device 104 or data processing system102. The digital component provider device 106 or components thereof canbe integrated with the data processing system 102 or executed at leastpartially by the data processing system 102. The digital componentprovider device 106 can include at least one logic device such as acomputing device having a processor to communicate via the network 105,for example with the computing device 104, the data processing system102, or the digital component provider device 106. The digital componentprovider device 106 can include at least one computation resource,server, processor, or memory. For example, the digital componentprovider device 106 can include a plurality of computation resources orservers located in at least one data center.

A digital component provider device 106 can provide audio, visual, ormultimedia-based digital components for presentation by the clientcomputing device 104 as an audio output digital component or visualoutput digital component. The digital component can be or include adigital content. The digital component can be or include a digitalobject. The digital component can include subscription-based content orpay-for content. A digital component can include a plurality of digitalcontent items. For example, a digital component can be a data streamfrom a streaming music service (e.g., the digital component providerdevice 106). The streamed digital component can include multiple songsas different digital content items. The digital components can includeor can be digital movies, websites, songs, applications (e.g.,smartphone or other client device applications), or other text-based,audio-based, image-based, or video-based content.

The digital component provider device 106 can provide the digitalcomponents to the client computing device 104 via the network 105 andbypass the data processing system 102. The digital component providerdevice 106 can provide the digital component to the client computingdevice 104 via the network 105 and data processing system 102. Forexample, the digital component provider device 106 can provide thedigital components to the data processing system 102, which can storethe digital components and provide the digital components to the clientcomputing device 104 when requested by the client computing device 104.

The data processing system 102 can include at least one computationresource or server. The data processing system 102 can include,interface, or otherwise communicate with at least one interface 110. Thedata processing system 102 can include, interface, or otherwisecommunicate with at least one remote digital assistant component 112.The remote digital assistant component 112 can include, interface, orotherwise communicate with at least one NLP component 114, at least onedirect action handler component 135, and a least one remote applicationlauncher 116. The data processing system 102 can include, interface, orotherwise communicate with at least one digital component selector 120.The data processing system 102 can include, interface, or otherwisecommunicate with at least one data repository 124. The at least one datarepository 124 can include or store, in one or more data structures ordatabases, deep links 126, policies 128, templates 130, or content data132. The data repository 124 can include one or more local ordistributed databases, and can include a database management

The components of the data processing system 102 can each include atleast one processing unit or other logic device such as a programmablelogic array engine or module configured to communicate with the databaserepository or database 124. The components of the data processing system102 can be separate components, a single component, or part of multipledata processing systems 102. The system 100 and its components, such asa data processing system 102, can include hardware elements, such as oneor more processors, logic devices, or circuits.

The data processing system 102 can include an interface 110. Theinterface 110 can be configured, constructed, or operational to receiveand transmit information using, for example, data packets. The interface110 can receive and transmit information using one or more protocols,such as a network protocol. The interface 110 can include a hardwareinterface, software interface, wired interface, or wireless interface.The interface 110 can facilitate translating or formatting data from oneformat to another format. For example, the interface 110 can include anapplication programming interface (“API”) that includes definitions forcommunicating between various components, such as software components.

The data processing system 102 can include an application, script, orprogram installed at the client computing device 104, such as a localdigital assistant 134 to communicate input audio signals to theinterface 110 of the data processing system 102 and to drive componentsof the client computing device to render output audio signals or visualoutput. The data processing system 102 can receive data packets, adigital file, or other signals that include or identify an input audiosignal (or input audio signals). The computing device 104 can detect theaudio signal via the transducer 140 and convert the analog audio signalto a digital file via an analog-to-digital converter. For example, theaudio driver 142 can include an analog-to-digital converter component.The pre-processor component 140 can convert the audio signals to adigital file that can be transmitted via data packets over network 105.

The remote digital assistant component 112 of the data processing system102 can execute or run an NLP component 114 to receive or obtain thedata packets including the input audio signal detected by the sensor 138of the computing device 104. The data packets can provide a digitalfile. The NLP component 114 can receive or obtain the digital file ordata packets comprising the audio signal and parse the audio signal. Forexample, the NLP component 114 can provide for interactions between ahuman and a computer. The NLP component 114 can be configured withtechniques for understanding natural language and enabling the dataprocessing system 102 to derive meaning from human or natural languageinput. The NLP component 114 can include or be configured withtechniques based on machine learning, such as statistical machinelearning. The NLP component 114 can utilize decision trees, statisticalmodels, or probabilistic models to parse the input audio signal. The NLPcomponent 114 can perform, for example, functions such as named entityrecognition (e.g., given a stream of text, determine which items in thetext map to names, such as people or places, and what the type of eachsuch name is, such as person, location (e.g., “home”), or organization),natural language generation (e.g., convert information from computerdatabases or semantic intents into understandable human language),natural language understanding (e.g., convert text into more formalrepresentations such as first-order logic structures that a computermodule can manipulate), machine translation (e.g., automaticallytranslate text from one human language to another), morphologicalsegmentation (e.g., separating words into individual morphemes andidentify the class of the morphemes, which can be challenging based onthe complexity of the morphology or structure of the words of thelanguage being considered), question answering (e.g., determining ananswer to a human-language question, which can be specific oropen-ended), or semantic processing (e.g., processing that can occurafter identifying a word and encoding its meaning in order to relate theidentified word to other words with similar meanings).

The NLP component 114 can convert the input audio signal into recognizedtext by comparing the input signal against a stored, representative setof audio waveforms (e.g., in the data repository 124) and choosing theclosest matches. The set of audio waveforms can be stored in datarepository 124 or other database accessible to the data processingsystem 102. The representative waveforms are generated across a largeset of users, and then may be augmented with speech samples from theuser. After the audio signal is converted into recognized text, the NLPcomponent 114 matches the text to words that are associated, for examplevia training across users or through manual specification, with actionsthat the data processing system 102 can serve. The NLP component 114 canconvert image or video input to text or digital files. The NLP component114 can process, analyze, or interpret image or video input to performactions, generate requests, or select or identify data structures.

The data processing system 102 can receive image or video input signals,in addition to, or instead of, input audio signals. The data processingsystem 102 can process the image or video input signals using, forexample, image interpretation techniques, computer vision, a machinelearning engine, or other techniques to recognize or interpret the imageor video to convert the image or video to a digital file. The one ormore image interpretation techniques, computer vision techniques, ormachine learning techniques can be collectively referred to as imagingtechniques. The data processing system 102 (e.g., the NLP component 114)can be configured with the imaging techniques, in addition to, orinstead of, audio processing techniques.

The NLP component 114 can obtain the input audio signal. From the inputaudio signal, the NLP component 114 can identify at least one request,at least one trigger keyword corresponding to the request, and one ormore entities. The request can indicate intent, digital components, orsubject matter of the input audio signal. The trigger keyword canindicate a type of action likely to be taken. For example, the NLPcomponent 114 can parse the input audio signal to identify at least onerequest to leave home for the evening to attend dinner and a movie. Thetrigger keyword can include at least one word, phrase, root or partialword, or derivative indicating an action to be taken. The intent can bean expressly stated intent that identifies an action to be taken or anagent to interact with. For example, the input audio signal thatincludes an express intent for a car service can be “Ok, get me a carservice to the go to the movies.” The intent can also be derived or notexpressly stated. For example, in the input audio signal that includes“Ok, I want to go to the movies” does not expressly request a carservice, but the NLP component 114 can derive the intent. For example,the trigger keyword “go” or “to go to” from the input audio signal canindicate a need for transport. In this example, the input audio signal(or the identified request) does not directly express an intent fortransport, however the trigger keyword indicates that transport is anancillary action to at least one other action that is indicated by therequest.

The NLP component 114 can parse the input audio signal to identify,determine, retrieve, or otherwise obtain the request and the triggerkeyword. For instance, the NLP component 114 can apply a semanticprocessing technique to the input audio signal to identify the triggerkeyword or the request. The NLP component 114 can apply the semanticprocessing technique to the input audio signal to identify a triggerphrase that includes one or more trigger keywords, such as a firsttrigger keyword and a second trigger keyword. For example, the inputaudio signal can include the sentence “Play my favorite song.” The NLPcomponent 114 can determine that the input audio signal includes atrigger keyword “play.” The NLP component 114 can determine that therequest is for the end user's favorite song (a digital component). TheNLP component 114 can identify an application identifier or a digitalcomponent provider device 106 identifier in the input audio signal. Theapplication identifier or digital component provider device 106identifier can indicate which application or digital component providerdevice 106 the end user would like to fill the request.

The data processing system 102 can execute or run an instance of thedirect action handler component 135. The direct action handler component135 can execute scripts or programs based on input received from the NLPcomponent 114. The service provider device 160 can provide the scriptsor programs. The service provider device 160 can make the scripts orprograms available to the data processing system 102 through an API orwebhooks. The direct action handler component 135 can determineparameters or responses to input fields and can package the data into anaction data structure. The action data structure can be provided to thedata processing system 102 through an API or webhooks. The direct actionhandler component 135 can transmit the action data structure to aservice provider device 160 for fulfillment or the data processingsystem 102 can fulfill the action data structure.

The direct action handler component 135 can generate or select, based onthe request or the trigger keyword identified in an input audio signal,data structures for the actions of a thread or conversation. Based onthe request parsed by the NLP component 114, the direct action handlercomponent 135 can determine to which of a plurality of service providercomputing devices 160 the message should be sent. The direct actionhandler component 135 can determine that the input audio signal includesa request for an explicit service provider device 160 (e.g., “Order acar with Car Service XYZ,” where the request specifically requests therequest be fulfilled by Car Service XYZ) or can select from a pluralityof service provider devices 160 can fulfill the request. The directaction handler component 135 can package the request into an action datastructure for transmission as a message to a service provider computingdevice 160. The action data structure can include information forcompleting the request. The information can be data that the serviceprovider device 160 uses to complete the request. Continuing the aboveexample for a car service request, the information can include a pick uplocation and a destination location. The direct action handler component135 can retrieve a template 130 from the repository 124 to determinewhich fields to include in the action data structure. The direct actionhandler component 135 can retrieve content from the repository 124 toobtain information for the fields of the data structure. The directaction handler component 135 can populate the fields from the templatewith that information to generate the data structure. The direct actionhandler component 135 can also populate the fields with data from theinput audio signal or previous input audio signals. The templates 130can be standardized for categories of service providers or can bestandardized for specific service providers. For example, ride sharingservice providers can use the following standardized template 149 tocreate the data structure: {client_deviceidentifier;authentication_credentials; pick_uplocation; destination_location;no_passengers; service_level}.

The direct action handler component 135 can be configured to expandresponses or entities contained in the responses. The direct actionhandler component 135 can expand entities that the NLP component 114identifies in the input audio signal. The direct action handlercomponent 135 can expand the entities to convert the entities into aformat that the service provider device 160 requires for a given fieldof its action data structures. The entities can include information thatmay be ambiguous or unclear to a service provider device 160. Forexample, when the service provider device 160 requested a streetaddress, the end user may provide an entity that is the proper name of alocation or business. The direct action handler component 135 canautomatically generate the expanded entity based on content orpreferences the data processing system 102 received from the clientcomputing device 104. The direct action handler component 135 cangenerate the expanded entity based on content or preferences the dataprocessing system 102 requests from the client computing device 104 in asubsequent audio-based input request. For example, the data processingsystem 102 can receive an input audio signal that includes “Ok, requesta car service to pick me up at home.” The NLP component 114 can identifythe term “home” as an entity that the service provider device 160 cannotrecognize. For example, the NLP component 114 can identify “home” as alocation entity; however, the location field in the action datastructure can require a street address, city, state, and zip code. Inthis example, the “home” location entity is not in the format requestedby the service provider device 160. If the end user of the clientcomputing device 104 previously provided the data processing system 102with the end user's home address, the direct action handler component135 can expand “home” into the format requested by field of the serviceprovider device's action data structure (e.g., {street_address: “123Main St.”, city: “Anytown”, state: “CA”}). If the end user did notpreviously provide the data processing system 102 with the end user'shome address, the data processing system 102 can generate and transmitan audio-based input request that requests the end user indicate aspecific address rather than “home.” Expanding the entity prior totransmitting the entity to the service provider device 160 can reducethe number of required network transmission because the service providerdevice 160 may not need to request clarifying or additional informationafter receiving the unexpanded entity.

The direct action handler component 135 can expand the entities based onan expansion policy 128 that is associated with the client computingdevice 104 that provided the input audio signal. The expansion policy128 can indicate whether the digital component provider device 128 haspermission to expand or convert the identified entity. For example, andcontinuing the above example, the direct action handler component 135can, based on the expansion policy 128, determine whether the directaction handler component 135 has access to convert the location entity“home” into a specific address.

The direct action handler component 135 can generating input requestsbased at least on the interface type of the client computing device 104and one or more fields of an action data structure for a serviceprovider device 160. The data processing system 102 can gatherinformation and populate the fields of a service provider device'saction data structure by transmitting a series of input requests to theclient computing device 104. For example, the data processing system 102can initiate a conversation-based data exchange with the clientcomputing device 104. The content of the input request can be defined bythe service provider device 160. For example, the service providerdevice 160 can define the dialog of the input requests that prompt theend user of the client computing device 104 to provide inputs to thedata processing system 102 (via the client computing device 104) suchthat the direct action handler component 135 can populate the fields ofthe action data structure. The service provider device 160 may have todevelop multiple instances of the dialogs to account for the differenttypes of client computing devices 104 that may interact with the dataprocessing system 102. The different types of client computing device104 may include different types of interfaces, which can also bereferred to as surfaces. For example, a first client computing device104 can include a voice-only interface, and a second client computingdevice 104 can include a screen-based interface.

The direct action handler component 135 can enable the service providerdevice 160 to provide only the action data structure the serviceprovider device 160 needs populated to fulfill a request. The directaction handler component 135, rather than the service provider device160, can generate or define the input requests based on the fields inthe action data structure. The direct action handler component 135 cangenerate or define the input requests based on the fields in the actiondata structure and a device type or interface type associate with theclient computing device 104. The input requests can be data objects thatcan include text, images, audio, videos, or any combination thereof.When executed or rendered by the client computing device 104, the clientcomputing device 104 generates output signals (via a screen, speaker, orother interface). For example, for a client computing device 104 thatincludes a touch screen, rendering the input request can cause theclient computing device 104 to display input boxes a user can select viathe touch screen. For a client computing device 104 that includes only avoice-based interface, rendering the input request can cause the clientcomputing device 104 to present audio prompts to the user via the clientcomputing device's speakers.

The direct action handler component 135 can populate one or more fieldsof the action data structure without generating an input request that istransmitted to the client computing device 104. The direct actionhandler component 135 can populate one or more fields of the action datastructure with content previously provide by the client computing device104 (or end user thereof). For example, if the action data structurerequests an address that the end user previously provided either whilesetting up the client computing device 104 (or an account associatedwith the client computing device 104) or in a prior input audio signal.

The data processing system 102 can execute or run an instance of theremote application launcher 116. The remote application launcher 116 canidentify, via the NLP component 114, keywords, tokens, terms, concepts,intents, or other information in the digital file or prior audio inputto identify one or more applications stored on or executable by theclient computing device 104 to fulfill the identified intent. The remoteapplication launcher 116 can parse log files of application installsprovided by the client computing device 104 to determine whichapplications are installed on the client computing device. For example,the NLP component 114 can determine that the intent for the input audiosignal “Ok, request a car to take me to the airport” is to request a carservice. When the client computing device 104 installs an applicationconfigured to fulfill one or more intents, the application can registerwith the data processing system 102. For example, the application canprovide the data processing system 102 with a manifest file thatindicates which intents the application can fulfill. The data processingsystem 102 can store an indication of the application and intent in thedatabase in association with a unique identifier associated with theclient computing device 104. Based on determining the intent of therequest, the remote application launcher 116 can search the manifestsstored in association with the unique identifier for the clientcomputing device 104 and select an application that is configured tofulfill the intent.

The remote application launcher 116 can generate a deep link. The remoteapplication launcher 116 can transmit the deep link to the clientcomputing device 104. The deep link can cause the client computingdevice 104 to launch or invoke the application that the remoteapplication launcher 116 selected to fulfill the intent. The dataprocessing system 102 can store templates of the deep-links associatedwith the applications executed by the client computing device 104 asdeep links 126 in the repository 124. The remote application launcher116 can populate the deep links with entities identified in the inputaudio signal, expanded entities, or other content the client computingdevice 104 (or user thereof) previously provided the data processingsystem 102. Once launched, the application can populate one or morefields in the application with the information or data populated intothe deep links. For example, continuing the above example, the remoteapplication launcher 116 can populate the deep link with the end usershome address and the airport address. Execution of the deep link by theapplication can, for example, cause the car service application toautomatically populate the start location field with the end user's homeaddress and the destination field with the address of the airport.

The data processing system 102 can execute or run an instance of thedigital component selector 120. The digital component selector 120 canselect a digital component that includes text, strings, characters,video files, image files, or audio files that can be processed by theclient computing device 104 and presented to the user via the display144 or the transducer 140 (e.g., speaker). Deep links can be an exampledigital component. Action data structures can be an example digitalcomponent.

The digital component selector 120 can select a digital component thatis responsive to the request identified by the NLP component 114 in theinput audio signal. The digital component selector 120 can select whichdigital component provider device 106 should or can fulfill the requestand can forward the request to the digital component provider device106. For example, the data processing system 102 can initiate a sessionbetween the digital component provider device 106 and the clientcomputing device 104 to enable the digital component provider device 106to transmit the digital component to the client computing device 104.The digital component selector 120 can request digital component fromthe digital component provider device 106. The digital componentprovider device 106 can provide digital components to the dataprocessing system 102, which can store the digital components in thedata repository 124. Responsive to a request for a digital component,the digital component selector 120 can retrieve the digital componentfrom the data repository 124.

The digital component selector 120 can select multiple digitalcomponents via a real-time content selection process. The digitalcomponent selector 120 can score and rank the digital components andprovide multiple digital components to the output merger component 120to allow the output merger component 120 to select the highest rankingdigital component. The digital component selector 120 can select one ormore additional digital components that are transmitted to a secondclient computing device 104 based on an input audio signal (or keywordsand requests contained therein). The one or more additional digitalcomponents can be transmitted to the second client computing device 104as part of or in addition to the authorization request. For example, ifthe input audio signal from the first client computing device 104includes a request for a digital component from a first digitalcomponent provider device 106, the digital component selector 120 canselect one or more digital components that correspond to one or morerelated digital component provider devices 106. In one illustrativeexample, the input audio signal can include a request to start astreaming music to a first client computing device 104. The digitalcomponent selector 120 can select additional digital components (e.g.,ads) that are associated with a different digital component providerdevice 106. The data processing system 102 can include the additionaldigital components in the authorization request to the second clientcomputing device 104. The additional digital components can inform anend user of additional or related digital component provider devices 106that could fulfill the request from the first client computing device104.

The digital component selector 120 can provide the digital componentselected in response to the request identified in the input audio signalto the computing device 104, or local digital assistant 134, orapplication executing on the computing device 104 for presentation.Thus, the digital component selector 120 can receive the content requestfrom the client computing device 104, select, responsive to the contentrequest, a digital component, and transmit, to the client computingdevice 104, the digital component for presentation. The digitalcomponent selector 120 can transmit, to the local digital assistant 134,the selected digital component for presentation by the local digitalassistant 134 itself or a third-party application executed by the clientcomputing device 104. For example, the local digital assistant 134 canplay or output an audio signal corresponding to the selected digitalcomponent.

The data repository 124 can store deep links 126 that can be stored inone or more data structures or data files. The deep links 126 can bestored in a table or database. The deep links 126 can include links,pointers, references, or other address or location information of anapplication that can be executed by the client computing device 104. Adeep link to an application can refer to a uniform resource locator orhyperlink that links to a specific resource, web content, application,or view within an application. The deep link can include the informationused to point to a particular resource or application, launch anapplication, or populate predetermined fields within the application.Deep links can include uniform resource identifiers (“URI”) that linksto a specific location within a mobile application in addition tolaunching the application. For example, opening, selecting, or executinga deep link on the client computing device 104 can cause the clientcomputing device 104 to open an application associated with the digitalcomponent provider device 106. The deep link fully or partiallypre-populates a registration or sign-up form within the openedapplication. In this example, to register for the subscription serviceoffered by the digital component provider device 106, the end user mayneed to only review the information entered into the form and thenselect a confirmation icon located on the loaded application page. Thedeep links 126 data structure can include a hash table that mapsapplication names or views of an application to a deep link.

The data repository 124 can store expansion policies 128 that can bestored in one or more data structures or data files. The expansionpolicies 128 can indicate what data, information, or entities can beexpanded. The data processing system 102 can store a different expansionpolicy 128 for each of the client computing devices 104 associated withthe data processing system 102.

The data repository 124 store content data 132 that can include, forexample, digital components provided by a digital component providerdevice 106 or obtained or determined by the data processing system 102to facilitate content selection. The content data 132 can include, forexample, digital components (or digital component object) that caninclude, for example, a content item, an online document, audio, images,video, multimedia content, or third-party content. The content data 132can include digital components, data, or information provided by theclient computing devices 104 (or end user thereof). For example, thecontent data 132 can include user preferences, user information storedby the user, or data from prior input audio signals.

The service provider device 160 can execute, include, or access aservice provider NLP component 161 and a service provide interface 162.The service provider NLP component 161 can be an instance of or performthe functions of the NLP component 114. For example, the data processingsystem 102 can forward an input audio signal to the service providerdevice 160 and the service provider NLP component 161 can process theinput audio signal. The service provider interface 162 can be aninstance of the interface 110. The service provider interface 162 can besoftware interface or a hardware interface. The service providerinterface 162 can be an API or set of webhooks.

FIG. 2 illustrates a block diagram of an example method 200 to generatevoice-activated threads in a networked computer environment. The methodcan include receiving an input audio signal (ACT 202). The method 200can include parsing the input audio signal (ACT 204). The method 200 caninclude selecting an action data structure (ACT 206). The method 200 caninclude expanding a response entity (ACT 208). The method can includepopulating the action data structure (ACT 210). The method 200 caninclude transmitting the digital component (ACT 212).

The method 200 can include can include receiving an input signal (ACT202). The method can include receiving, by an NLP component executed bya data processing system, the input signal. The input signal can be aninput audio signal that is detected by a sensor at a first client deviceand transmitted to the data processing system. The sensor can be amicrophone of the first client device. For example, a digital assistantcomponent executed at least partially by a data processing system thatincludes one or more processors and memory can receive the input audiosignal. The input audio signal can include a conversation facilitated bya digital assistant. The conversation can include one or more inputs andoutputs. The conversation can be audio based, text based, or acombination of audio and text. The input audio signal can include textinput, or other types of input that can provide conversationalinformation. The data processing system can receive the audio input fora session corresponding to the conversation.

The method 200 can include parsing the input signal (ACT 204). The NLPcomponent of the data processing system can parse the input signal toidentify a request. The NLP component can identify at least one entityin the input signal. The request can be an intent or request that can befulfilled by one or more service provider devices. The request can be apart of a conversational phrase. For example, the request can be “Ok,order a car to take me home.” The entities identified by the NLPcomponent can be phrases or terms in the request that map to inputfields or types the service provider device requests when fulfilling arequest. For example, the service provider device providing the carservice may request a current location input field and a destinationinput field. Continuing the above example, the NLP component can map theterm “home” to the destination input field.

The method 200 can include selecting an action data structure (ACT 206).The data processing system can select the action data structure based onthe request parsed from the input signal. The data processing system canselect the action data structure based on the service provider devicethat can fulfill the request. The action data structure can be a datastructure or object that is created by the service provider device. Theservice provider device can provide the action data structure to thedata processing system. The action data structure can indicate fields,data, or information that the service provider device uses to fulfillrequests. The service provider device can flag one or more of the fieldsto request that the data processing system expand the entity returnedfor that field. When a field is flagged for expansion, the dataprocessing system can design and generate conversation-based dataexchanges with the client computing device 104 to retrieve informationor data for the flagged field rather than the service provider device160 designing the conversation-based data exchange.

The method 200 can include expanding the response entity (ACT 208). Thedata processing system can determine the entity mapped to the inputfield needs to be expanded if the entity is not in a format specified bythe service provider device. Continuing the above example, the NLPcomponent can determine “home” is the entity mapped to a destination.The direct action handler component can determine to update the actiondata structure to include the entity “home” in a destination field. Thedirect action handler component can determine the format of the responseentity does not match the format of the destination field. For example,the destination field can have the format of an object that requests astreet address, city, state, and zip code. Detecting a mismatch betweenthe format of the response entity and the format of the field, the dataprocessing system can expand the entity to a street address, city,state, and zip code format. For example, the data processing system canlook up the address the end user provided the data processing system asthe end user's “home” address. The data processing system can check thatthe format of the response entity does not match the format of thedestination field and rather than forwarding the response to the serviceprovider device, the data processing system can expand the entity. Thedata processing system can automatically expand the entity because thedata processing system knows that the response provided by the user isnot in the correct format and the response could cause the agent of theservice provider device to fail. For example, the service providerdevice may not be able to understand the entity “home.” If the dataprocessing system determines the entity is not in the correct format andthe data processing system cannot expand the entity into the properformat, the data processing system can transmit an input request to theclient computing device that can prompt the user of the client computingdevice for additional information. The data processing system can expandthe entity based on an expansion policy. The expansion policy canindicate whether the data processing system has permission to expand theterm or can indicate what end user or client computing device provideddata can be included in an expanded entity.

The data processing system can expand the entity based on a request froma service provider device. For example, the data processing system cangenerate a first action data structure with the unexpanded entity. Thedata processing system can transmit the first action data structure tothe service provider device for processing to fulfill the request. Theservice provider device can return the action data structure (or aportion thereof) to the data processing system if the service providerdevice cannot process or understand the data in on or more of the actiondata structure's fields. For example, the service provider device canattempt to process the “home” entity in the destination field and thenrequest the data processing system expand the “home” entity after theservice provider device determines that it cannot process or understandthe entity.

The method 200 can include populating the action data structure (ACT210). The direct action handler component can populate the action datastructure with the expanded entity. The direct action handler componentcan populate the action data structure with the entity. For example, theaction data structure can be an object into which the entity or expandedentity is stored. Populating the action data structure can also bereferred to update the action data structure.

The method 200 can include transmitting the action data structure (ACT212). The data processing system can transmit the populated action datastructure to the service provider device. Upon receipt of the actiondata structure, the service provider device can fulfill the request orrequest additional information from the data processing system or clientcomputing device.

FIG. 3 illustrates a block diagram of an example data flows betweencomponents of the system 100 illustrated in FIG. 1. FIG. 3 illustratesthe data flows during an example implementation of the method 200illustrated in FIG. 3. The example includes the expansion of an entityidentified in an input signal. The example can include transmitting oneor more data flows between the data processing system 102, clientcomputing device 104, and the service provider device 160.

At a first time point, the service provider device 160 can transmit afirst data flow 301 to the data processing system 102. The first dataflow 301 can include an action data structure template. The action datastructure template can indicate to the data processing system 102 whatfields the service provider device 160 may require to fulfill one ormore intents or services. For example, the first data flow 301 caninclude an action data structure template for requesting a car service.In the example illustrated in FIG. 3, the action data structure templateincludes {origin: “ ”, dest: “ ”}, which indicates that the serviceprovider device 160 may need an origin location and a destinationlocation when a car service is requested.

The client computing device 104 can transmit a data flow 302 to the dataprocessing system 102 to invoke a service provided by the serviceprovider device 160. The data flow 302 can include an input audiosignal. The input audio signal can, in the example illustrated in FIG.3, include the phrase “Ok, order a car service.” Based on the invocationof the data flow 302, the data processing system 102 can transmit aninput request, such as an audio-based input request, to the clientcomputing device 104 in data flow 303. Rendering or presentation of theinput request illustrated in FIG. 3 at the client computing device 104can cause the client computing device 104 to present through a speakerat the client computing device 104 “Where would you like to go?” The enduser can respond to the input request. For example, the end user canspeak “from my office to the airport” in response to the input request.The client computing device 104 can via the sensor 138 (e.g., amicrophone) detect the end user's response, digitize the response, andtransmit the response to the data processing system 102 as an inputaudio signal within the data flow 304.

The data processing system 102 can parse the input audio signal andidentify one or more response entities in the input audio signal. Thedata processing system 102 can identify response entities that arevague. The response entities can be terms or phrases that the NLPcomponent 114 maps to one or more fields of the action data structure.For example, as illustrated in FIG. 3, the action data structureincludes two fields that each require locations as inputs. The dataprocessing system 102 can map the term “office” and “airport” asresponse entities that map to the formats associated with the fields ofthe action data structure. Based on an expansion policy associated withthe client computing device 104, the data processing system 102 canexpand the response entities into a format required by the fields of theaction data structure. In the example illustrated in FIG. 3, the“office” response entity is expanded to “123 Main St.” and the term“airport” is expanded to “323 5th St.” The action data structure withthe expanded response entities can be transmitted to the serviceprovider device 160 in the data flow 305.

FIG. 4 illustrates a block diagram of an example method 400 to generatevoice-activated threads in a networked computer environment. The methodcan include receiving an input audio signal (ACT 402). The method 400can include parsing the input audio signal (ACT 404). The method 400 caninclude determining a plurality of candidate service provider devices(ACT 406). The method 400 can include selecting a service providerdevice (ACT 408). The method 400 can include generating a digitalcomponent (ACT 410). The method 400 can include transmitting the digitalcomponent (ACT 412).

As set forth above, and also referring to FIG. 1, among others, themethod 400 can include can include receiving an input signal (ACT 402).The method can include receiving, by an NLP component executed by a dataprocessing system, the input signal. The input signal can be an inputaudio signal that is detected by a sensor at a first client device andtransmitted to the data processing system. The sensor can be amicrophone of the first client device. For example, a digital assistantcomponent executed at least partially by a data processing system thatincludes one or more processors and memory can receive the input audiosignal. The input audio signal can include a conversation facilitated bya digital assistant. The conversation can include one or more inputs andoutputs. The conversation can be audio based, text based, or acombination of audio and text. The input audio signal can include textinput or other types of input that can provide conversationalinformation. The data processing system can receive the audio input fora session corresponding to the conversation.

The method 400 can include parsing the input signal (ACT 404). The NLPcomponent of the data processing system can parse the input signal toidentify a request and at least one entity. The request can be an intentor request that can be fulfilled by one or more service providerdevices. The request can be a conversational phrase. For example, therequest can be “Ok, order a car to take me home.” The entitiesidentified by the NLP component can be phrases or terms in the requestthat map to input fields or types the service provider device requestswhen fulfilling a request. For example, the service provider deviceproviding the car service may request a current location input field anda destination input field. Continuing the above example, the NLPcomponent can map the term “home” to the destination input field.

The method 400 can include determining or identifying a plurality ofcandidate service provider devices (ACT 406). Each of the plurality ofcandidate service provider devices can be configured to fulfill therequest identified in the input audio signal. Each of the candidateservice provider devices can be associated with a respective applicationthat is installed on the client computing device. For example, each ofthe candidate service provider devices can be car service providers thatprovide applications that an end user can install on their clientcomputing device (e.g., smart phone) to request a car from the carservice provider. The data processing system can maintain a databasethat indicates which applications are installed on the client computingdevice. For example, the applications can be installed through anapplication store provided by the data processing system and the dataprocessing system can update an entry in the data repository whenapplications are installed or removed from the client computing device.Determining or identifying which service provider device can fulfill therequest can include identifying a data processing system that caninclude a plurality of devices, identifying a webhook address of theservice provider device, an IP address of the service provider device, adomain name of the service provider device, or an API of the serviceprovider device.

If the data processing system determines that the no applicationassociated with one of the candidate service provider devices isinstalled on the client computing device, the data processing system canselect a service provider device from the candidate service providerdevices and generate an action data structure rather than a digitalcomponent (e.g., deep link). The data processing system can populate theaction data structure with entities or expanded entities identified inthe input signal. The data processing system can transmit the actiondata structure to the selected service provider device to fulfill therequest.

The method 400 can include selecting a service provider device (ACT408). The data processing system can select the service provider devicefrom the plurality of service provider devices. The data processingsystem can select the service provider device based on a network latencybetween the data processing system and each of the candidate serviceprovider devices. For example, if the latency between a specific serviceprovider device and the data processing system is high, the dataprocessing system can determine that the candidate service providerdevice is experiencing network or other computational problems and thatnew request should not be transmitted to the specific candidate serviceprovider device. The data processing system can select the serviceprovider device from the candidate service provider devices based on aselection made by an end user of the client computing device. Forexample, the data processing system can transmit an input request to theclient computing device that when rendered by the client computingdevice presents a list of the candidate service provider devices. Theend user can provide a selection to the data processing system throughan input audio signal transmitted from the client computing device. Thedata processing system can select the service provider device based on aperformance of the candidate service provider devices. The performancecan be a computational performance or a service rating. For example, theservice rating can be generated by user reviews after interacting withthe specific service provider device. The data processing system canselect the service provider device based on which candidate serviceprovider device has the highest service rating.

The method 400 can include generating a digital component (ACT 410). Themethod 400 can include generating, by the action handler component, adigital component that can include an indication of the applicationassociated with the service provider device. The digital component canalso include the first entity. The digital component, when executed orrendered by the client computing device, can cause the client computingdevice to launch the application associated with the service providerdevice. The digital component can be a deep link. The digital componentcan be an action data structure. The application can be used to fulfillthe request in the input audio signal. For example, and continuing theabove example, when executed, the digital component can launch the carrequesting application on the end user's smart phone. The digitalcomponent can include one or more fields. The direct action handlercomponent can populate the entity or an expanded version of the entityinto one or more fields of the digital component.

The method 400 can include transmitting the digital component to thefirst client computing device (ACT 412). The data processing system cantransmit the digital component to the client computing device inresponse to receiving the input audio signal. When the client computingdevice receives the digital component (e.g., a deep link), the digitalcomponent can be rendered on a screen of the client computing device.The end user can activate the client computing device by selecting thedeep link. Activation of the deep link can cause the client computingdevice to open the application associated with the selected serviceprovider device. The deep link can also populate fields (e.g., textfields) of the open application with the entities populated into thedigital component.

FIG. 5 illustrates a block diagram of example data flows betweencomponents of the system 100 in illustrated in FIG. 1. FIG. 5illustrates the data flows during an example implementation of themethod 400 illustrated in FIG. 4. The example includes the expansion ofan entity identified in an input signal and the fulfillment of an intentwith deep links. The data flows can be between a client computing device104, a data processing system 102, a first service provider device160(1), and a second service provider device 160(2).

As illustrated in FIG. 5, and also referring to FIG. 1 among others, theclient computing device 104 can transmit a first input audio signal tothe data processing system 102 in a first data flow 501. In the exampleillustrated in FIG. 5, the input audio signal includes the phrase “Ok,get me a car service home.” The data processing system 102 receive thedata flow and determine that the input audio signal includes a requestor intent for a car service. The data processing system 102 can access alog file that indicates which applications are installed on the clientcomputing device 104. The log file can also indicate which intents orrequest types each of the applications can fulfill. For example, theclient computing device 104 can include a first application that isassociated with the service provider device 160(1), which can be a firstcar service provider and a second application that is associated withthe service provider device 160(2), which can be a second car serviceprovider. The data processing system 102 can select one of the serviceprovider device 160(1) and the service provider device 160(2) to fulfillthe request in the input audio signal.

The data processing system 102 can select a deep link template that isassociated with the selected service provider device 160 from the deeplinks 126 stored in the data repository 124. For example, in the exampleillustrated in FIG. 5, the data processing system 102 can select theservice provider device 160(1). The data processing system 102 cangenerate the deep link and populate the fields of the deep link withentities or expanded entities the NLP component 114 identified in theinput audio signal. For example, the entity “home” in the input audiosignal can be expanded to “123 Main St.” The data processing system 102can transmit the deep link to the client computing device 104 in dataflow 502. When the client computing device 104 renders or executes thedeep link, the deep link can cause the client computing device 104 toopen or execute the application identified in the deep link. Forexample, the deep link transmitted to the client computing device 104 inthe data flow 502 can cause the client computing device 104 to open theapplication associated with the service provider device 160(1).

Execution of the deep link can also cause the data from the fields ofthe deep link to be automatically populated into one or more fields ofthe open application. For example, when the client computing device 104opens the application associated with the service provider device160(1), the deep link can cause “123 Main St.” to be populated into thedestination field of the application. The client computing device 104,via the application associated with the service provider device 160(1),can send a data flow 503 to the service provider device 160(1) tofulfill the request in the input audio signal of the data flow 501.

The data processing system can generate voice-activated threads in anetworked computer environment. The data processing system can receive afirst input audio signal that is detected by a sensor of a first clientcomputing device. The data processing system can parse the first inputaudio signal to identify a first request. The data processing system canselect a first action data structure based on the first request. Thefirst action data structure can be associated with a first serviceprovider device. The data processing system can transmit a firstaudio-based input request to the first client based at least on a fieldin the first action data structure. The data processing system canreceive a second input audio signal detected by the sensor of the firstclient computing device that is generated in response to the firstaudio-based input request. The data processing system can parse thesecond input audio signal to identify a response entity in the secondinput audio signal. The data processing system can expand the responseentity based on an expansion policy that is associated with the firstclient computing device. Expanding the response entity can includeconverting the response entity into format associated with the field inthe first action data structure. The data processing system can populatethe expanded response entity into the field of the first action datastructure. The data processing system can transmit the first action datastructure to the first service provide to fulfill the first request.

FIG. 6 is a block diagram of an example computer system 600. Thecomputer system or computing device 600 can include or be used toimplement the system 100, or its components such as the data processingsystem 102. The data processing system 102 can include an intelligentpersonal assistant or voice-based digital assistant. The computingsystem 600 includes a bus 605 or other communication component forcommunicating information and a processor 610 or processing circuitcoupled to the bus 605 for processing information. The computing system600 can also include one or more processors 610 or processing circuitscoupled to the bus for processing information. The computing system 600also includes main memory 615, such as a random access memory (RAM) orother dynamic storage device, coupled to the bus 605 for storinginformation, and instructions to be executed by the processor 610. Themain memory 615 can be or include the data repository 124. The mainmemory 615 can also be used for storing position information, temporaryvariables, or other intermediate information during execution ofinstructions by the processor 610. The computing system 600 may furtherinclude a read-only memory (ROM) 620 or other static storage devicecoupled to the bus 605 for storing static information and instructionsfor the processor 610. A storage device 625, such as a solid-statedevice, magnetic disk or optical disk, can be coupled to the bus 605 topersistently store information and instructions. The storage device 625can include or be part of the data repository 124.

The computing system 600 may be coupled via the bus 605 to a display635, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 630, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 605 for communicating information and command selections to theprocessor 610. The input device 630 can include a touch screen display635. The input device 630 can also include a cursor control, such as amouse, a trackball, or cursor direction keys, for communicatingdirection information and command selections to the processor 610 andfor controlling cursor movement on the display 635. The display 635 canbe part of the data processing system 102, the client computing device104 or other component of FIG. 1, for example.

The processes, systems and methods described herein can be implementedby the computing system 600 in response to the processor 610 executingan arrangement of instructions contained in main memory 615. Suchinstructions can be read into main memory 615 from anothercomputer-readable medium, such as the storage device 625. Execution ofthe arrangement of instructions contained in main memory 615 causes thecomputing system 600 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory615. Hard-wired circuitry can be used in place of or in combination withsoftware instructions together with the systems and methods describedherein. Systems and methods described herein are not limited to anyspecific combination of hardware circuitry and software.

Although an example computing system has been described in FIG. 6, thesubject matter including the operations described in this specificationcan be implemented in other types of digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them.

For situations in which the systems discussed herein collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's location), or to control whether or how toreceive content from a content server or other data processing systemthat may be more relevant to the user. In addition, certain data may beanonymized in one or more ways before it is stored or used, so thatpersonally identifiable information is removed when generatingparameters. For example, a user's identity may be anonymized so that nopersonally identifiable information can be determined for the user, or auser's geographic location may be generalized where location informationis obtained (such as to a city, postal code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about him or her and usedby the content server.

The subject matter and the operations described in this specificationcan be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. The subject matter described in thisspecification can be implemented as one or more computer programs, e.g.,one or more circuits of computer program instructions, encoded on one ormore computer storage media for execution by, or to control theoperation of, data processing apparatuses. Alternatively, or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. While acomputer storage medium is not a propagated signal, a computer storagemedium can be a source or destination of computer program instructionsencoded in an artificially generated propagated signal. The computerstorage medium can also be, or be included in, one or more separatecomponents or media (e.g., multiple CDs, disks, or other storagedevices). The operations described in this specification can beimplemented as operations performed by a data processing apparatus ondata stored on one or more computer-readable storage devices or receivedfrom other sources.

The terms “data processing system,” “computing device,” “component,” or“data processing apparatus” encompass various apparatuses, devices, andmachines for processing data, including by way of example a programmableprocessor, a computer, a system on a chip, or multiple ones, orcombinations of the foregoing. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or anASIC (application specific integrated circuit). The apparatus can alsoinclude, in addition to hardware, code that creates an executionenvironment for the computer program in question, e.g., code thatconstitutes processor firmware, a protocol stack, a database managementsystem, an operating system, a cross-platform runtime environment, avirtual machine, or a combination of one or more of them. The apparatusand execution environment can realize various different computing modelinfrastructures, such as web services, distributed computing and gridcomputing infrastructures. For example, the interface 110, digitalcomponent selector 120, the direct action handler component 135, remotelauncher application 116, or NLP component 114 and other data processingsystem 102 components can include or share one or more data processingapparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, softwareapplication, app, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages,declarative or procedural languages, and can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, object, or other unit suitable for use in a computingenvironment. A computer program can correspond to a file in a filesystem. A computer program can be stored in a portion of a file thatholds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs (e.g., components of the data processing system 102)to perform actions by operating on input data and generating output. Theprocesses and logic flows can also be performed by, and apparatuses canalso be implemented as, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). Devices suitable for storing computer programinstructions and data include all forms of non-volatile memory, media,and memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

The subject matter described herein can be implemented in a computingsystem that includes a back end component, e.g., as a data server, orthat includes a middleware component (e.g., an application server), orthat includes a front end component (e.g., a client computer having agraphical user interface or a web browser through which a user caninteract with an implementation of the subject matter described in thisspecification), or a combination of one or more such back end,middleware, or front end components. The components of the system can beinterconnected by any form or medium of digital data communication(e.g., a communication network). Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system such as system 100 or system 600 can includeclients and servers. A client and server are generally remote from eachother and typically interact through a communication network (e.g., thenetwork 105). The relationship of client and server arises by virtue ofcomputer programs running on the respective computers and having aclient-server relationship to each other. In some implementations, aserver transmits data (e.g., data packets representing a digitalcomponent) to a client device (e.g., for purposes of displaying data toand receiving user input from a user interacting with the clientdevice). Data generated at the client device (e.g., a result of the userinteraction) can be received from the client device at the server (e.g.,received by the data processing system 102 from the client computingdevice 104 or the digital component provider device 106).

While operations are depicted in the drawings in a particular order,such operations are not required to be performed in the particular ordershown or in sequential order, and all illustrated operations are notrequired to be performed. Actions described herein can be performed in adifferent order.

The separation of various system components does not require separationin all implementations, and the described program components can beincluded in a single hardware or software product. For example, the NLPcomponent 114, direct action handler component 135, remote launcherapplication 116, or the digital component selector 120, can be a singlecomponent, app, or program, or a logic device having one or moreprocessing circuits, or part of one or more servers of the dataprocessing system 102.

Having now described some illustrative implementations, it is apparentthat the foregoing is illustrative and not limiting, having beenpresented by way of example. In particular, although many of theexamples presented herein involve specific combinations of method actsor system elements, those acts and those elements may be combined inother ways to accomplish the same objectives. Acts, elements andfeatures discussed in connection with one implementation are notintended to be excluded from a similar role in other implementations orimplementations.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including” “comprising” “having” “containing” “involving”“characterized by” “characterized in that” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular may also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein mayalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element may include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein may be combined with any otherimplementation or embodiment, and references to “an implementation,”“some implementations,” “one implementation” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described in connectionwith the implementation may be included in at least one implementationor embodiment. Such terms as used herein are not necessarily allreferring to the same implementation. Any implementation may be combinedwith any other implementation, inclusively or exclusively, in any mannerconsistent with the aspects and implementations disclosed herein.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall the described terms. For example, a reference to “at least one of‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and‘B’. Such references used in conjunction with “comprising” or other openterminology can include additional items.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded to increase the intelligibility of the drawings, detaileddescription, and claims. Accordingly, neither the reference signs northeir absence has any limiting effect on the scope of any claimelements.

The systems and methods described herein may be embodied in otherspecific forms without departing from the characteristics thereof. Forexample, the computing device 104 can generate the packaged data objectand forward it to the third-party application when launching theapplication. The foregoing implementations are illustrative rather thanlimiting of the described systems and methods. Scope of the systems andmethods described herein is thus indicated by the appended claims,rather than the foregoing description, and changes that come within themeaning and range of equivalency of the claims are embraced therein.

The invention claimed is:
 1. A system to generate voice-activatedthreads in a networked computer environment, comprising: a naturallanguage processor component executed by a data processing system to:receive first input audio signal detected by a sensor of a first clientcomputing device; parse the first input audio signal to identify a firstrequest; parse the first input audio signal to identify a responseentity in the first input audio signal; a digital component selectorexecuted by the data processing system to select a digital componentprovided by a digital component provider device based on the firstrequest, and transmit the digital component to the first clientcomputing device, wherein the digital component comprises an indicationof an application associated with a service provider device and theresponse entity; an action handler component executed by the dataprocessing system to select a first action data structure based on thefirst request, the first action data structure associated with a firstservice provider device; the action handler component to: convert theresponse entity based on an expansion policy associated with the firstclient computing device and into a format associated with a field in thefirst action data structure; store the response entity in the formatassociated with the field in the first action data structure; and aninterface of the data processing system to transmit the first actiondata structure to the first service provider device to fulfill the firstrequest.
 2. The system of claim 1, comprising: the digital componentselector to transmit the digital component to a local digital assistantexecuting on the first client computing device.
 3. The system of claim1, comprising: the digital component selector to transmit the digitalcomponent to a third-party application executed by the first clientcomputing device.
 4. The system of claim 1, comprising: the digitalcomponent selector to select the digital component via a real-timecontent selection process that comprises determines a score and a rankfor each of a plurality of digital components to cause selection of ahighest scoring and ranking digital component of the plurality ofdigital components.
 5. The system of claim 1, comprising: the digitalcomponent selector to select the digital component configured to fulfillthe first request, wherein the digital component is different from thefirst action data structure.
 6. The system of claim 1, comprising: thedigital component selector to select the digital component associatedwith a second service provider device different from the first serviceprovider device, wherein the digital component is configured to fulfillthe first request.
 7. The system of claim 1, comprising: the actionhandler component to transmit a first audio-based input request to thefirst client computing device based at least on a second field in thefirst action data structure; the natural language processor component toreceive a second input audio signal detected by the sensor of the firstclient computing device and generated in response to the firstaudio-based input request; the natural language processor component toparse the second input audio signal to identify a second response entityin the second input audio signal; and the action handler component topopulate the second field in the first action data structure with thesecond response entity.
 8. The system of claim 1, comprising: thenatural language processor component to receive a second input audiosignal detected by the sensor of the first client computing device; thenatural language processor component to parse the second input audiosignal to identify a second request; the action handler component toselect a second action data structure based on the second request, thesecond action data structure associated with the first service providerdevice; the natural language processor component to parse, the secondinput audio signal to identify a second response entity in the secondinput audio signal; and the action handler component to determine not toexpand the second response entity based on the expansion policyassociated with the first client computing device.
 9. The system ofclaim 1, comprising: the natural language processor component to map theresponse entity to the field in the first action data structure.
 10. Thesystem of claim 1, comprising: the action handler component to determinethat the response entity comprises a second format not associated withthe format associated with the field in the first action data structure;and the action handler component to expand the response entity based onthe response entity having the second format not associated with theformat associated with the field in the first action data structure. 11.A method of generating voice-activated threads in a networked computerenvironment, comprising: receiving, by a data processing system, firstinput audio signal detected by a sensor of a first client computingdevice; parsing, by the data processing system, the first input audiosignal to identify a first request; parsing, by the data processingsystem, the first input audio signal to identify a response entity inthe first input audio signal; selecting, by the data processing system,a digital component provided by a digital component provider devicebased on the first request, wherein the digital component comprises anindication of an application associated with a service provider deviceand the response entity; transmitting, by the data processing system,the digital component to the first client computing device; selecting,by the data processing system, a first action data structure based onthe first request, the first action data structure associated with afirst service provider device; converting, by the data processingsystem, the response entity based on an expansion policy associated withthe first client computing device and into a format associated with afield in the first action data structure; storing, by the dataprocessing system, the response entity in the format associated with thefield in the first action data structure; and transmitting, by the dataprocessing system, the first action data structure to the first serviceprovider device to fulfill the first request.
 12. The method of claim11, comprising: transmitting, by the data processing system, the digitalcomponent to a local digital assistant executing on the first clientcomputing device.
 13. The method of claim 11, comprising: transmitting,by the data processing system, the digital component to a third-partyapplication executed by the first client computing device.
 14. Themethod of claim 11, comprising: selecting, by the data processingsystem, the digital component via a real-time content selection processthat comprises determines a score and a rank for each of a plurality ofdigital components to cause selection of a highest scoring and rankingdigital component of the plurality of digital components.
 15. The methodof claim 11, comprising: selecting, by the data processing system, thedigital component configured to fulfill the first request, wherein thedigital component is different from the first action data structure. 16.The method of claim 11, comprising: selecting, by the data processingsystem, the digital component associated with a second service providerdevice different from the first service provider device, wherein thedigital component is configured to fulfill the first request.
 17. Themethod of claim 11, comprising: transmitting, by the data processingsystem, a first audio-based input request to the first client computingdevice based at least on a second field in the first action datastructure; receiving, by the data processing system, a second inputaudio signal detected by the sensor of the first client computing deviceand generated in response to the first audio-based input request;parsing, by the data processing system, the second input audio signal toidentify a second response entity in the second input audio signal; andpopulating, by the data processing system, the second field in the firstaction data structure with the second response entity.
 18. The method ofclaim 11, comprising: receiving, by the data processing system, a secondinput audio signal detected by the sensor of the first client computingdevice; parsing, by the data processing system, the second input audiosignal to identify a second request; selecting, by the data processingsystem, a second action data structure based on the second request, thesecond action data structure associated with the first service providerdevice; parsing, by the data processing system, the second input audiosignal to identify a second response entity in the second input audiosignal; and determining, by the data processing system, not to expandthe second response entity based on the expansion policy associated withthe first client computing device.
 19. The method of claim 11,comprising: mapping, by the data processing system, the response entityto the field in the first action data structure.
 20. The method of claim11, comprising: determining, by the data processing system, that theresponse entity comprises a second format not associated with the formatassociated with the field in the first action data structure; andexpanding, by the data processing system, the response entity based onthe response entity having the second format not associated with theformat associated with the field in the first action data structure.